Add CRC64 checksum check #18682
Add CRC64 checksum check #18682
Conversation
| GetBlockChecksumRequest request) { | ||
| // Non-batched mode | ||
| if (request.getBlockIdsCount() < mChecksumBatchSize) { | ||
| long startTs = CommonUtils.getCurrentMs(); |
There was a problem hiding this comment.
Does the client just send requests without doing logical processing? Would it be better to put this part of the logic processing on a higher level?
There was a problem hiding this comment.
i moved the logic to the util
| * Heap buffer pool. | ||
| */ | ||
| public class NioHeapBufferPool { | ||
| private static final TreeMap<Integer, LinkedList<ByteBuffer>> BUF_POOL = new TreeMap<>(); |
There was a problem hiding this comment.
The current memory pool is only used by CRC, and the application will not be recycled. Will it occupy core business resources? I think CRC verification is a low-frequency non-core business that does not require high efficiency. Should I directly apply for heap memory temporarily? That’s it
There was a problem hiding this comment.
i added a non-pooled option
| import java.nio.ByteBuffer; | ||
| import java.util.zip.Checksum; | ||
|
|
||
| public class CRC64 implements Checksum { |
There was a problem hiding this comment.
Can crc64 not be used as a standard library function? Because different ufs crcs have different algorithms. For example, oss and cos are crc64, s3 is crc32, and hdfs has its own verification. Can we directly connect the verification algorithm of each ufs? Which one is better?
| @@ -946,6 +946,10 @@ public FileInfo getFileInfo(long fileId) | |||
| @Override | |||
| public FileInfo getFileInfo(AlluxioURI path, GetStatusContext context) | |||
| throws FileDoesNotExistException, InvalidPathException, AccessControlException, IOException { | |||
| if (context.getOptions().hasDirectUfsAccess() && context.getOptions().getDirectUfsAccess()) { | |||
There was a problem hiding this comment.
On 2.x, there is still no link from the master to the ufs request. Does this break the current architecture? Regardless of worker or proxy, they all get the mount id and attributes from the master and request ufs independently.
| } | ||
| crc64.update(bf.array(), Math.toIntExact(bytesRead)); | ||
| int permits = Math.toIntExact(Math.max(1, bytesRead / 1024)); | ||
| if (mChecksumCalculationRateLimiter.isPresent()) { |
There was a problem hiding this comment.
Is it better to obtain the token before reading?
There was a problem hiding this comment.
it's hard to know how many bytes read there so i decide to obtain the token after that.
|
Automated checks report:
Some checks failed. Please fix the reported issues and reply |
|
Automated checks report:
All checks passed! |
|
alluxio-bot, merge this please. |
|
merge failed: |
|
alluxio-bot, merge this please. |
|
alluxio-bot, cherry-pick this to branch-2.10 please. |
|
Auto cherry-pick unsuccessful Failed to setup local git for auto cherry-pick 1a3fbc9 cannot be auto cherry-picked to branch branch-2.10. Please cherry-pick this manually. |
 To check the file integrity Add a crc64 check cli pr-link: Alluxio#18682 change-id: cid-4eb62e603591f3235b92e4191c5b511b534f6fc3
 To check the file integrity Add a crc64 check cli pr-link: Alluxio#18682 change-id: cid-4eb62e603591f3235b92e4191c5b511b534f6fc3
Cherry-pick of existing commit. orig-pr: #18682 orig-commit: 1a3fbc9 orig-commit-author: elega <[email protected]> pr-link: #18688 change-id: cid-4eb62e603591f3235b92e4191c5b511b534f6fc3
What changes are proposed in this pull request?
Why are the changes needed?
To check the file integrity
Does this PR introduce any user facing changes?
Add a crc64 check cli